GEM: Glare or Gloom, I Can Still See You – End-to-End Multi-Modal Object Detection

Deep neural networks designed for vision tasks are often prone to failure when they encounter environmental conditions not covered by the training data. Single-modal strategies insufficient sensor fails acquire information due malfunction or its design limitations. Multi-sensor configurations known provide redundancy, increase reliability, and crucial in achieving robustness against asymmetric failures. To address issue of changing lighting degradation object detection, we develop a multi-modal 2D detector, propose deterministic stochastic sensor-aware feature fusion strategies. The proposed mechanisms driven estimated measurement reliability values/weights. Reliable detection harsh is essential applications such as self-driving vehicles human-robot interaction. We also new “r-blended” hybrid depth modality RGB-D sensors. Through extensive experimentation, show that outperform existing state-of-the-art methods on FLIR-Thermal dataset, obtain promising results SUNRGB-D dataset. additionally record RGB-Infra indoor namely L515-Indoors, demonstrate methodologies highly effective variety conditions.